EDIT: I can’t change the title, but this appears to be more serious than a bad update. Read on....

https://www.neowin.net/news/y2k22-bug-microsoft-rings-in-the-new-year-by-breaking-exchange-servers-all-around-the-world/

——————————————————

Just wondering if any other Exchange admins had their new year’s celebration interrupted due to the “Microsoft Filtering Management Service” being stopped and reports of issues with mail flow?

In the application event logs, I see a bunch of errors from FIPFS service which say: Cannot convert “220101001” to long

If I look back further in the logs, it appears like it all started happening when the “MS Filtering Engine Update” process received the “220101001” update version just over an hour ago at 7:57pm EST.

EDIT: I’ve tried forcing it to check for another update, but it returned “MS Filtering Engine Update process has not detected any new scan engine updates”. ... I’ve temporarily disabled anti-malware scanning, to restore mail flow for now.

TL DR; Microsoft released a bad update for Exchange 2016 and 2019. Disabling OR bypassing anti-malware filtering will restore mail flow in the interim

UPDATE: according to @ceno666 the issue also seems to occur with the 220101002 update version as well. Could be related to, what I’m dubbing, the “Y2K22” bug. Refer to the comment from JulianSiebert about the “signed long” here: https://techcommunity.microsoft.com/t5/exchange-team-blog/december-2021-exchange-server-cumulative-updates-postponed/bc-p/3049189/highlight/true#M31885 The “long” type allows for values up to 2,147,483,647. It appears that Microsoft uses the first two numbers of the update version to denote the year of the update. So when the year was 2021, the first two numbers was “21”, and everything was fine. Now that it’s 2022 (GMT), the update version, converted to a “long” would be 2,201,01,001 - - which is above the maximum value of the “long” data type. @Microsoft: If you change it to an ‘unsigned long’, then the max value is 4,294,967,295 and we’ll be able to sleep easy until the year 2043!

UPDATE: Microsoft has confirmed disabling the malware filtering is the correct course of action for now (workaround to restore mail flow). While new signatures and engine updates have been released, they don’t seem to fix the issue. We’ll continue to wait for an official response from Microsoft. At least we have a third-party filtering/scanning solution in front of Exchange.

UPDATE: If you still have mail flow delays after disabling the malware filter, check your transport rules; you might have a rule that is trying to check attachments; reference this comment for information on finding the correct transport rule: https://www.reddit.com/r/sysadmin/comments/rt91z6/exchange_2019_antimalware_bad_update/hqtt5ib/

UPDATE: Reddit user u/MarkDePalma created a custom script to roll back to 2021 and reportedly allows you to re-enable all malware filtering while we wait for a patch from Microsoft. PROCEED AT YOUR OWN RISK, ‘John Titor’, haha. https://blog.markdepalma.com/?p=810

UPDATE, 01/01 14:39 EST (19:39 GMT): Microsoft has released a statement here: https://techcommunity.microsoft.com/t5/exchange-team-blog/email-stuck-in-transport-queues/ba-p/3049447

UPDATE, 01/02 01:45 EST (06:45 GMT): Microsoft has released a fix for the “Y2K22 Exchange Bug” which requires action to be taken on each Exchange server in your environment. Some system administrators report this fix can take around 30 minutes to run, which could increase depending on how many people are trying to simultaneously download the update from the Microsoft servers. Interestingly, this fix includes a change to the format of the problematic update version number; the version number now starts with “21” again, to stay within the limits of the ‘long’ data type, for example: “2112330001”. So, Happy December 33, 2021! 😉 https://techcommunity.microsoft.com/t5/exchange-team-blog/email-stuck-in-transport-queues/ba-p/3049447

EDIT: If after applying the fix mentioned above, your queues may not clear and you may see a new FIPFS error with Event ID 2203, A FIP-FS Scan process returned error 0x84004003 ... Msg: Scanning Process caught exception ... Unknown error 2214608899. Failed to meet engine bias criteria (Available) for filter type (Malware). To fix this issue, restart the Microsoft Filtering Management Service: Restart-Service FMS -Force